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Introduction 

One  reason  for  the  recent  resurgence 
of  interest  in  neural  network-like  compu¬ 
tational  models  has  been  the  prospect  of 
compact  and  fast  implementations  of  these 
networks  in  integrated  circuit  form. 
While  analog  implementations  offer  con¬ 
siderable  advantages  with  regard  to  speed 
and  density,  their  precision  and  noise 
immunity  are  important  concerns .  Some 
researchers  (e.g. ,  Mead  and  coworkers) 
have  built  analogues  of  biological  struc¬ 
tures  for  early  sensory  processing,  and 
they  have  emphasized  that  tolerance  of 
noisy  and  imprecise  components  is  a  nat¬ 
ural  emergent  feature  of  these  networks. 
However,  the  ways  in  which  "higher"  or 
"cognitive"  functions  might  be  learned 
and  computed  with  such  components  for  the 
most  part  remains  unknown.  In  addition, 
learning  itself  remains  problematic  in 
analog  circuitry.  Means  proposed  for 
long-term,  modifiable  analog  weight  stor¬ 
age  (e.g.,  floating-gate  MOS  devices)  are 
sensitive,  difficult  to  control,  and  of 
limited  precision. 

We  have  chosen  to  implement  a  model 
of  olfactory  processing  proposed  by 
Granger,  Lynch,  and  Ambros-Ingerson , 
which  we  believe  to  be  an  instructive 
paradigm  for  computation  in  a  learning 
system  with  low-precision  weights  and 
weight  changes.1'2  The  model  has  been 
shown  capable  of  performing  a  heirarch- 
ical  clustering  of  vectors  on  its  input 
space.  This  capability  is  of  potential 
interest  for  a  range  of  applications, 
from  automatic  target  recognition  (ATR) 
to  surveillance  and  detection.  The  net¬ 
work  requires  only  coarse-valued  weights 
(three  to  five  bits  resolution)  and  its 
operation  relies  on  the  statistical  prop¬ 
erties  of  large  assemblages  of  sparsely 
interconnected  ..curo>'e,  rather  than  high 
precision  processing.  In  addition, 
clustering  capability  is  acquired  by  an 
unsupervised  coactivity-based  learning 
rule  that  requires  only  increments  and 


admits  a  simple  parallel  implementation. 
The  fact  that  the  network  can  learn  to 
extract  structure  from  its  input  environ¬ 
ment  in  an  unsupervised  fashion  is  of 
general  interest  for  the  advancement  of 
autonomous  or  "smart"  systems  with  a 
broad  range  of  defense  and  space  applica¬ 
tions. 

The  Model 

The  interested  reader  is  referred  to 
the  work  of  Granger  et  al.  for  details  of 
the  olfactory  model.1'2  Herein  we  give 
an  abbreviated,  qualitative  summary  of 
the  model  in  engineering  terms.  The 
system  accepts  a  vector  input  with  com¬ 
ponents  of  like  algebraic  sign.  This 
input  is  subject  to  a  form  of  normaliza¬ 
tion  which  constrains  total  sut-mgn iti’de 
of  the  processed  signals  to  on  the  order 
of  20%  of  maximum.  Each  normalized  com¬ 
ponent  is  then  "thermometer-coded,"  or 
quantized  and  represented  by  the  number 
of  active  two-state  cells  within  a  group 
("glomerulus")  of  such  cells  .  associated 
with  that  component.  The  outputs  of 
these  ("mitral")  cells  project  via  a 
randomly  interconnected,  sparse  (i.e.,  on 
the  order  of  10%  full)  weight  matrix  to 
a  second  set  of  (“piriform")  cells,  which 
are  arranged  in  winner-take-all  subnet¬ 
works.  The  most  strongly  stimulated 
piriform  cell  in  each  subnetwork  activ¬ 
ates,  and  the  sparse  pattern  of  activity 
of  the  piriform  cells  represents  the 
output  of  the  system. 

The  active  piriform  cells  subsequent¬ 
ly  send  inhibitory  feedback  to  the  glo¬ 
meruli  via  correl at iona 1 iy-developed 
(Hebbian)  interconnections,  so  that  inhi¬ 
bition  is  strongest  for  those  input  com¬ 
ponents  most  responsible  for  the  piriform 
activation.  The  input  is  resampled, 
combined  with  this  inhibition,  and  re¬ 
normalized,  resulting  in  a  pattern  of 
mitral  activity  in  which  initially  large 
components  are  squelched  and  secondary 
components  are  expressed  more  strongly. 
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The  new  pattern  of  mitral  activity  is  fed 
forward  to  the  piriform  cells  and  results 
in  a  new  pattern  of  piriform  activity. 
This  cycle  is  typically  repeated  a  number 
of  times. 

When  the  system  is  in  the  learning 
mode,  the  weights  of  interconnections 
between  active  mitral  and  winning  piri¬ 
form  cells  are  incremented  at  each  cycle 
of  activity.  As  a  consequence  of  this 
learning,  the  piriform  output  codes  for 
input  vectors  which  are  sufficiently 
similar,  or  clustered  in  the  input  space, 
tend  to  develop  a  high  degree  of  overlap 
or  become  identical.  In  addition,  with 
the  multiple  sampling  and  feedback  inhi¬ 
bition,  the  inputs  are  clustered  into 
succeedingly  finer  subclusters  or  cate¬ 
gories  (indicated  by  identical  or  nearly 
identical  piriform  output  codes)  based  on 
secondary  and  tertiary  characteristics. 

Implementation 

We  propose  a  direct  analog  CMOS  im¬ 
plementation  of  this  model.  This  network 
would  operate  synchronously  with  regard 
to  the  resampling  cycle  required  for 
heirarchical  clustering,  with  feedforward 
and  feedback  subphases.  However,  compu¬ 
tations  of  neuronal  states  would  be  ana¬ 
log,  asynchronous,  and  carried  out  in 
parallel.  Current  mode  design  techniques 
have  been  employed.  Floating-gate  weight 
storage  and  on-chip,  parallel  learning 
are  proposed. 

The  normalization  required  by  the 
model  can  be  achieved  with  a  circuit 
analogous  to  vector  automatic  gain  con¬ 
trol,  with  saturating  nonlinearity  app¬ 
lied  to  each  component.  Thermometer¬ 
coding  is  performed  by  a  circuit  analog¬ 
ous  to  the  input  stage  of  a  parallel  A/D 
converter.  On  the  piriform  side,  winner- 
take-all  subnetworks  are  implemented 
using  circuits  with  global  feedback.3 

For  "synaptic"  weights,  we  propose 
the  use  of  single  floating-gate  transis¬ 
tors  whose  transconductance  is  modulated 
by  charge  cn  the  floating  gate.  We 
propose  to  implement  the  sparse,  random 
weight  matrix  with  a  one-to-one 
correspondence  of  the  number  of  weighting 
elements  to  number  of  synapses  in  the 
model,  with  mask-programmable  connection 
of  input  and  output  lines  allowing  estab¬ 
lishment  of  pseudorandom  connectivity. 

To  implement  feedback  inhibition  of 
the  bulb  by  the  piriform,  we  propose  a 
time-duplex  scheme.  Because  the  correla- 
tionally-developed  feedback  interconnec¬ 
tions  arise  as  a  direct  consequence  of 
the  given  feedforward  connectivity,  the 
same  inhibitory  effect  can  be  obtained  by 


using  the  transpose  of  the  sparse  weight 
matrix  to  compute  the  inhibition.  Physi¬ 
cally,  a  single  weight  matrix  would  be 
driven  bidirectionally  in  two  phases,  to 
successively  compute  both  feedforward 
excitation  and  feedback  inhibition.  To 
allow  this  bidirectionality,  interface  is 
made  to  the  weight  matrix  via  type-II 
current  conveyors.4 

For  individual  weights,  the  control 
logic  for  the  coactivity-based  learning 
rule  corresponds  to  a  simple  AND  func¬ 
tion;  taken  in  parallel  it  may  be  regard¬ 
ed  as  a  Boolean  outer  product.  This  can 
be  implemented  on  crossbars  running 
through  the  weight  matrix  using  simple 
switches  which  are  controlled  by  the 
neuron  states  and  which  route  programming 
voltages  to  writing  circuitry  for  the 
floating-gate  weights. 

The  building-block  circuits  required 
for  the  implementation  described  above 
have  been  fabricated  using  the  MOSIS  and 
Orbit  Semiconductor  Foresight  prototyping 
services.  DC  functionality  was  estab¬ 
lished  experimentally  foi  most,  with  AC 
or  transient  response  estimated  from 
SPICE  simulations.  We  present  circuit 
designs  and  test  and  simulation  results 
in  poster  illustrations,  including  re¬ 
sults  from  a  variety  of  candidates  for 
floating-gate  weight  circuits  which  were 
fabricated  in  the  standard  processes.  A 
simple  scheme  to  compensate  for  the 
strong  nonlinearity  of  the  floating  gate 
charging  mechanisms  is  described.  Plans 
for  a  network  with  on  the  order  of  35,000 
weights  are  also  detailed. 
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